Eecient Covariance Matrix Methods for Bayesian Gaussian Processes and Hoppeld Neural Networks

Covariance matrices are important in many areas of neural modelling. In Hop eld networks they are used to form the weight matrix which controls the autoassociative properties of the network. In Gaussian processes, which have been shown to be the in nite neuron limit of many regularised feedforward neural networks, covariance matrices control the form of Bayesian prior distribution over function space. This thesis examines interesting modi cations to the standard covariance matrix methods to increase functionality or e ciency of these neural techniques. Firstly the problem of adapting Gaussian process priors to perform regression on switching regimes is tackled. This involves the use of block covariance matrices and Gibbs sampling methods. Then the use of Toeplitz methods is proposed for Gaussian process regression where sampling positions can be chosen. A comparison is made between Hop eld weight matrices, and sample covariances. This allows work on sample covariances to be used to estimate the eigenvalue structure of standard Hop eld weight matrices. This structure of Hop eld weight matrices enables the development of e cient incremental and local learning techniques for Hop eld networks. A new Hop eld learning rule is introduced. It is shown both empirically and mathematically that such learning methods give a higher capacity than the usual Hebb rule. Furthermore it is shown that this rule is more able to deal with the storage of correlated patterns. The basins of attraction of this learning rule are examined empirically. It is found that they are larger, rounder and more evenly distributed than those of the Hebb rule. Lastly we show using iterated function sequence methods that the learning rule acts as a palimpsest (or forgetful) learning rule, and does not su er from catastrophic forgetting. 3